Experience in Extending Query Engine for Continuous Analytics

 Experience in Extending Query Engine for Continuous Analytics Qiming Chen, Meichun Hsu HP Laboratories HPL-2010-44 In-Database Stream Processing Combining data warehousing and stream processing technologies has great potential in offering low-latency data-intensive analytics. Unfortunately, such convergence has not been properly addressed so far. The current generation of stream processing systems is in general built separately from the data warehouse and query engine, which can cause significant overhead in data access and data movement, and is not able to take advantage of the functionalities already offered by the existing data warehouse systems. In this work we tackle some hard problems not properly addressed previously in integrating stream analytics capability into the existing query engine. We define an extended SQL query model that unifies queries over both static relations and dynamic streaming data, and develop techniques to generalize query engines to support the unified model. We propose the cut-and rewind query execution model to allow a query to be applied to stream data by converting the latter into a sequence of "chunks", and executing the query over each chunk sequentially without shutting the query instance down between chunks;, we also propose the cycle-based transaction model to support Continuous Querying with Continuous Persisting (CQCP) with cycle-based isolation and visibility. We have prototyped our approach by extending the PostgreSQL. This work has resulted in a new kind of tightly integrated, highly efficient system with the advanced stream processing capability as well as the full DBMS functionality. We demonstrate the system with the popular Linear Road benchmark, and report the performance. By leveraging the more mature codebase of a query engine to the maximal extent, we can significantly reduce the engineering investment needed for developing the streaming technology. Providing this capability on HP SeaQuest parallel analytics engine is work in progress. External Posting Date: May 21, 2010 [Fulltext] Approved for External Publication Internal Posting Date: May 21, 2010 [Fulltext] Copyright 2010 Hewlett-Packard Development Company, L.P. Experience in Extending Query Engine for Continuous Analytics Qiming Chen Meichun Hsu HP Labs HP Labs Palo Alto, California, USA Palo Alto, California, USA Hewlett Packard Co. Hewlett Packard Co. [email protected] [email protected]